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Abstract 


The  DoD  is  a  large,  bureaucratic,  rule-intensive  organization  that  may  not  be 
suited  well  for  its  environment.  Building  upon  prior  research  of  acquisition  centralization 
and  knowledge  dynamics,  we  employ  computational  methods  to  assess  the  behavior 
and  performance  of  different  organizational  designs  in  varying  environments.  Our 
results  reinforce  Contingency  Theory  and  suggest  particular  characteristics  of  different 
acquisition  environments  make  one  organizational  form  relatively  more  or  less 
appropriate  than  another.  Practically,  answers  to  our  research  questions  have  direct 
and  immediate  application  to  acquisition  leaders  and  policy  makers.  Theoretically,  we 
generalize  to  broad  classes  of  organizations  and  prescribe  a  novel  set  of  organizational 
design  guides. 

Keywords:  Computational  Organizational  Theory,  VDT,  project  simulation, 
Defense  Acquisition,  centralization,  bureaucracy. 
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Executive  Summary 


The  DoD  is  a  large,  bureaucratic,  rule-intensive  organization  that  may  not  be 
suited  well  for  its  environment.  Building  upon  prior  research  on  acquisition  centralization 
and  knowledge  dynamics,  we  employ  computational  methods  to  assess  the  behavior 
and  performance  of  different  organizational  designs  in  varying  environments.  In  this 
study,  we  use  the  VDT  modeling  environment  to  represent  and  emulate  the  behavior  of 
an  acquisition  organization.  We  then  conceptualize  and  model  two  alternate  acquisition 
organizations,  manipulating  key  factors  of  their  organizational  designs.  We  subject 
them  to  two  environmental  contexts,  routine  and  stressed,  comparing  their  performance 
in  terms  of  cost,  schedule  and  risk.  In  routine  circumstances,  our  experimental 
decentralized  and  holon-type  organizations  out-perform  typical  hierarchies  in  measures 
of  cost  and  schedule.  Under  high  stress  from  task  uncertainty,  noise,  and  error 
probability,  our  decentralized  and  holon-type  organizations  completed  their  same 
project  work  volume  as  well,  faster,  and  for  less  cost  than  their  centralized  counterpart. 

In  both  environments,  however,  our  less  formal  organization  structures  yield  a  higher 
project-quality  risk.  Our  findings  are  similar  to  those  of  other  VDT  researchers  who  find 
the  relationship  between  organizational  performance  improvements  and  increasing 
project  risk  from  decentralization  in  environments  of  uncertainty  (Kim  &  Burton,  2002) 
and  worker  knowledge  (Nissen  &  Buettner,  2004).  They  offer  an  extension  of 
Contingency  Theory  to  include  risk  as  a  dependent  variable  for  organizational  structures 
and  project  outcomes.  Our  results  reveal  the  same  relational  patterns  of  performance 
capabilities  among  the  three  organizational  designs  and  across  differing  stress 
environments.  They  underscore  complex  interactions  between  organizational  design 
factors,  and  suggest  fundamental  tension  and  decision  tradeoffs  between  important 
performance  measures  such  as  project  cost,  schedule  and  quality/risk.  As  such,  they 
reinforce  Contingency  Theory  and  suggest  particular  characteristics  of  different 
acquisition  environments  that  make  one  form  relatively  more  or  less  appropriate  than 
another.  The  results  provide  several  implications  for  managerial  practices  and 
application  of  organization  theory  regarding  the  relationships  between  organizational 
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structure  and  performance.  Perhaps  for  the  first  time — or  at  least  to  an  extent 
unachievable  heretofore — we  show  how  managers  can  gain  fundamental  insights  into 
the  inherent  project  tradeoffs  in  advance  of  making  project  decisions.  Understanding 
when  the  bureaucracy  is  relatively  beneficial  and  how  this  rigid  organizational  form  can 
negatively  influence  project  cost  (and  positively  impact  project  risk)  is  important  for 
acquisition  practitioners  today.  The  apparent  implications  are  that  adopting  a 
decentralized  structure  in  accordance  with  contingency  theory  may  not  lead  to  higher 
unit  performance,  since  it  might  instead  produce  poorer  project  quality.  But  it  is 
insufficient  to  only  assume  that  more  bureaucracy  alleviates  risk  with  attendant  costs,  or 
that  managers  must  simply  choose  either  fast  and  cheap,  or  better  quality  results.  Our 
work  begins  to  shape  a  new  hypothesis:  that  perhaps  there  is  an  optimal  organizational 
design  solution,  relative  to  cost,  duration  and  risk.  If  managers  can  ascertain  early  on 
the  criticality  (and  tolerable  level)  of  project-quality  risk,  they  can  perhaps  select  along  a 
continuum  the  level  of  organizational  hierarchy  and  centralization  needed  to  control 
project  outcomes.  Or,  reframing  the  question,  how  much  will  added  bureaucracy  cost  to 
alleviate  risk?  Practically,  answers  to  our  research  questions  have  direct  and  immediate 
application  to  acquisition  leaders  and  policy  makers.  Theoretically,  we  generalize  to 
broad  classes  of  organizations  and  prescribe  a  novel  set  of  organizational  design 
guides. 
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Introduction 


Acquisition  is  big  business.  The  US  Department  of  Defense  (DoD)  alone  routinely 
executes  eleven-figure  budgets  for  research,  development,  procurement  and  support  of 
weapon  systems,  for  instance.  Acquisition  is  also  a  rule-intensive  business.  In  addition 
to  myriad  laws  governing  federal  acquisition  in  the  US,  a  plethora  of  regulations 
specify — in  great  detail  often — how  to  accomplish  the  planning,  review,  execution  and 
oversight  of  Government  acquisition  programs,  large  and  small,  sole-source  and 
competitive,  military  and  commercial  (Dillard,  2003).  Due  in  great  part  to  the  large  size 
and  many  rules  associated  with  Defense  acquisition  in  particular,  the  organizations 
responsible  for  DoD  acquisition  activities  tend  to  be  large  and  rule-intensive 
themselves — reflecting  the  kinds  of  centralized,  formalized,  specialized  and  oversight¬ 
intensive  forms  that  correspond  to  the  classic  Machine  Bureaucracy  from  Organization 
Theory  (e.g.,  see  Mintzberg,  1979).  The  problem  is  this  classic  organizational  structure 
is  known  well  to  be  exceptionally  poor  at  responding  to  change.  In  the  context  of  military 
transformation,  such  a  problem  should  be  clear  and  compelling.  Arguably,  one  or  more 
superior  organizational  approaches  must  be  available  to  replace  the  current  acquisition 
organization.  But  which,  if  any,  is  most  appropriate?  On  what  basis  should  acquisition 
leaders  and  policy  makers  choose  between  competing  organizational  forms?  What 
evidence  supports  claims  of  superiority  for  one  organizational  approach  versus 
another?  Questions  such  as  these  are  difficult  to  answer  through  most  research 
methods  employed  today  to  study  acquisition  organizations  (e.g.,  case  study,  survey, 
action). 

The  bureaucratic  nature  of  the  DoD  Acquisition  Organization  did  not  emerge 
recently,  nor  did  it  materialize  by  design.  Rather,  it  reflects  the  cumulative  accretion  of 
laws,  regulations,  rules  and  hierarchical  levels  over  considerable  time.  If  only  the 
organization  could  be  changed  and  evaluated — say  through  assessment  of  four 
alternate  organizational  structures — then  one  could  assess  the  relative  performance  of 
the  new  organizational  designs  versus  the  current  form  and  recommend  transformation 
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toward  the  best  performer.  But,  clearly  the  set  of  problems  and  actors  in  the  changed 
organizations  would  differ  from  those  associated  with  the  original  and  with  one  another; 
that  is,  there  is  no  way  to  impose  controls  over  such  a  study  (e.g.,  internal  validity  is 
compromised).  This  is  one  reason  why  so  many  acquisition  research  projects  produce 
so  little  new  knowledge.  Alternatively,  such  controls  can  be  imposed  easily  through 
laboratory  experimentation.  Yet,  the  simplified  nature  and  laboratory  context  of 
experiments  fail  to  capture  the  size,  scope  and  complexity  of  the  acquisition 
organization  (e.g.,  external  validity  is  compromised).  This  is  another  reason  why  so 
many  acquisition  research  projects  produce  so  little  new  knowledge.  However,  by 
combining  the  best  features  of  laboratory  experimentation  (e.g.,  experimental  controls) 
with  field  methods  (e.g.,  large-scale  and  complex  behaviors),  one  can  design  and 
conduct  a  study  of  acquisition  organizations  that  reflects  both  internal  and  external 
validity. 

This  is  the  approach  of  computational  experimentation:  using  sophisticated  and 
validated  computer  models  of  organizations  to  assess  the  behavior  and  performance  of 
different  organizational  designs.  Computational  Organization  Theory  (COT;  see  Carley 
&  Prietula,  1994)  provides  a  set  of  methods  and  tools  to  enable  this  approach.  In 
particular,  using  the  methods  and  tools  associated  with  the  Virtual  Design  Team  (VDT) 
Research  Group  at  Stanford,  computational  models  of  organizations  are  driven  by  well- 
accepted  organization  theory  and  are  validated  by  extensive  and  repeated  field  studies. 
This  validation  provides  considerable  confidence  that  computational  results  reflect  the 
likely  behaviors  and  performance  of  the  acquisition  organizations  they  model  and 
emulate. 

The  research  described  in  this  article  involves  the  application  of  VDT  methods 
and  tools  to  study  acquisition  organizations.  In  particular,  we  model  and  simulate  the 
behavior  of  organizations  associated  with  major  defense  acquisition  programs  in  the 
DoD.  We  provide  both  answers  to  and  insights  into  how  such  organizations  can  be 
changed  to  improve  performance.  Some  of  the  key  organizational  design  variables  of 
interest  pertain  to  the  bureaucratic  nature  of  the  organization  and  follow  from  recent 
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research  to  investigate  centralization  (Dillard,  2003).  For  instance,  factors  such  as 
centralization,  formalization,  specialization,  hierarchical  layers  and  the  like  can  be 
manipulated — individually  as  well  as  in  combination — under  controlled  and  replicated 
conditions  to  assess  the  performance  of  acquisition  organizations  in  different  forms. 

This  follows  recent,  complementary  research  using  computational  organization  theory  in 
the  domain  of  military  command  and  control  (Nissen  &  Buettner,  2004;  Nissen,  2005a, 
b).  Considerations  such  as  the  number,  frequency  and  level  of  acquisition  reviews, 
adaptability  and  flexibility  of  acquisition  organizations,  and  risk-versus-project-duration 
of  acquisition  programs  are  primary  in  this  study.  The  key  research  question  is:  How 
can  organizations  responsible  for  major  acquisition  programs  be  redesigned  to  improve 
performance? 

The  significance  of  this  approach  is  twofold.  First,  answers  to  the  research 
question  have  direct  and  immediate  application  to  acquisition  leaders  and  policy 
makers.  Such  answers  address  a  serious  and  immediate  problem,  revealing  insights 
into  the  behaviors  of  major  acquisition  organizations  that  are  too  complex  and  dynamic 
to  be  understood  well  or  directly.  They  illuminate  the  kinds  of  changes  acquisition 
organizations  can  make  to  balance  competing  performance  measures  (e.g.,  adaptability 
&  flexibility  vs.  project  risk  &  duration).  They  can  explain — in  a  theoretically  grounded 
manner — many  different  cases  of  acquisition  success  as  well  as  failure.  They  can  also 
provide  overarching  theory  to  help  promote  the  former  and  obviate  the  latter  in  future 
acquisition  programs. 

Second,  this  research  project  demonstrates  the  efficacy  of  a  new  approach  to 
studying  acquisition  organizations.  It  enables  leaders,  policy  makers  and  analysts  to 
answer  “how  much”  questions  such  as:  How  much  centralization,  formalization  and 
specialization  is  best?  What  fraction  of  commercial  off-the-shelf  equipment  would  be 
ideal?  What  level  of  concurrency  between  development  and  production  provides  the 
best  combination  of  cost,  schedule,  performance  and  risk?  Such  questions  are  not 
answered  well  today  in  terms  of  acquisition  organizations.  This  leaves  acquisition 
decision  makers  today  with  no  reliable  means  to  address  such  questions. 
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The  balance  of  the  article  begins  with  a  focused  review  of  the  literature  relevant 
to  this  study.  We  follow  with  discussion  of  our  research  design  and  description  of  the 
computation  model  developed  to  represent  and  emulate  the  acquisition  organization. 
The  article  turns  then  to  discuss  results  of  our  computational  experiments.  Conclusions, 
implications  and  recommendations  for  future  research  close  the  article,  along  with  a  rich 
set  of  references  for  deeper  exploration  into  the  research  on  which  this  article  builds 
and  contributes.  We  also  include  two  appendices  to  provide  details  of  our  computational 
models. 
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Background 


This  focused  review  of  the  literature  relevant  to  our  study  is  organized  into  three 
parts:  1)  the  acquisition  organization,  2)  organization  theory,  and  3)  computational 
experimentation. 

The  Acquisition  Organization 

Of  particular  interest  to  the  authors  is  the  realm  of  DoD  program  management, 
where  research  and  development  dollars  are  expended  to  invent  or  advance  warfighting 
capability.  While  US  weaponry  is  considered  some  of  the  best  in  the  world,  the  major 
acquisition  projects  to  acquire  them  are  often  fraught  with  cost  and  schedule  growth. 
They  even  fail  at  times  to  meet  specifications  or  to  provide  the  capabilities  desired. 

Since  implementation  of  the  Goldwater-Nichols  Act  legislation  in  the  late  1980s,  major 
defense  acquisition  organizations  (e.g.,  program  management  offices)  have  operated 
under  a  four-tiered  decision  structure. 

For  major  acquisitions,  the  current  policy  makes  clear  that  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics  is  the  Milestone  Decision  Authority 
responsible  for  the  overall  program:  Described  in  the  DODI  5000.1 : 

3.4  The  Milestone  Decision  Authority  (MDA)  is  the  designated  individual  with  overall 
responsibility  for  a  program.  The  MDA  shall  have  the  authority  to  approve  entry  of  an 
acquisition  program  into  the  next  phase  of  the  acquisition  process  and  shall  be 
accountable  for  cost,  schedule,  and  performance  reporting  to  higher  authority,  including 
Congressional  reporting.  (USD(AT&L),  2003) 

And  three  levels  down  the  hierarchy,  Program  Managers  (PMs)  are  described  as: 

3.5.1  the  designated  individual  with  responsibility  for  and  authority  to  accomplish 
program  objectives  for  development,  production,  and  sustainment  to  meet  the  user's 
operational  needs.  The  PM  shall  be  accountable  for  credible  cost,  schedule,  and 
performance  reporting  to  the  MDA.  (USD(AT&L),  2003) 

Thus,  the  Program  Manager  and  Milestone  Decision  Authority  share  responsibility  for 
development  and  oversight  of  a  program.  Further  guidance  under  the  DoD  Instruction 
5000.1  provides: 
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4. 3. 1.1  There  is  no  one  best  way  to  structure  an  acquisition  program  to  accomplish  the 
objective  of  the  Defense  Acquisition  System.  MDAs  and  PMs  shall  tailor  program 
strategies  and  oversight,  including  documentation  of  program  information,  acquisition 
phases,  the  timing  and  scope  of  decision  reviews  and  decision  levels  to  fit  the  particular 
conditions  of  that  program,  consistent  with  applicable  laws  and  regulations  and  the  time- 
sensitivity  of  the  capability  need.  (USD(AT&L),  2003) 

However,  while  the  wording  above  might  indicate  that  the  MDA  and  PM  plan 
jointly  or  collaborate  in  some  way  on  program  strategy,  there  are,  in  fact,  both  a 
Component  Acquisition  Executive  and  Program  Executive  Officer  in  the  hierarchy 
between  them,  and  direct  communication  between  MDA  and  PM  is  infrequent.  The  four 
tiers  of  major  program  command  and  control  and  typical  grade/ranks  of  positions  are 
shown  in  Figure  1 . 


Defense  Acquisition  Board  Chairman 
Political  Appointee  -  SES 
Milestone  Decision  Authority 
for  25  Major  Programs 

Undersecretary  of  Defense 

For  Acquisition,  Technology,  &  Logistics 


Defense  Acquisition  Board  Member 
Political  Appointee  -  SES 
Milestone  Decision  Authority 
for  lesser  Programs 


Undersecretary  of  Service 

For  Research,  Development,  &  Acquisition 


Defense  Acquisition  Board  Member 
Oversees  a  functional  area  or 
commodity  area  of  3  to  9  programs 


Flag  Officer  or  SES 


Defense  Acquisition  Board  Advisor 
Manages  one  or  more  programs 
Oversees  a  functional  area  teams 


Officer  or  GS-15/SES 


Program 

Manager 

0-6/ Flag 


Program 

Executive 

Officer 


Component 

Acquisition 

Executive 


Defense 

Acquisition 

Executive 


Figure  1.  DoD  Decision  Hierarchy  for  Major  Defense  Acquisition  Programs 

(adapted  from  DAU,  2004) 

MDA  PMs  lead  Program  Management  Offices  (PMOs).  PMOs  vary  greatly  in 
size.  A  typical  range  of  government-assigned  workers  is  generally  between  50  and  100 
individuals  dedicated  to  the  day-to-day  efforts.  An  expanding  network  of  other 
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government  agency  players,  multi-tier  industry  contractors,  and  other  participants  can 
multiply  this  figure  many  times  (Dillard,  2004).  While  all  stakeholders  represent  different 
parts  of  the  enterprise,  here  we  refer  to  this  central  government  organizational  entity — 
the  government  PMO — as  the  acquisition  organization. 

At  the  PMO  level,  several  alternatives  for  the  organization  exist.  In  most  cases, 
the  offices  are  comprised  of  permanently  assigned  “core”  personnel,  and  temporarily 
assigned  co-located  “matrix”  personnel  on  loan  from  commodity  systems  commands. 
These  are  personnel  typically  arrayed  by  functional  area  within  the  PMO  (as  shown  in 
Figure  2).  A  significant  number  of  on-site  support  contract  personnel  may  be  present  as 
well. 


PROGRAM  OFFICE  ORGANIZATION  STRUCTURE 


Figure  2.  Typical,  Matrixed  Program  Management  Office  Structure 

(adapted  from  DAU,  2004) 

Somewhat  less  formally,  programs  also  organize  internally  in  ad  hoc  teams 
oriented  on  specific  areas  of  each  project.  This  stems  largely  from  DoD  initiatives  over 
the  last  10  years  to  implement  Integrated  Product  and  Process  Development  (IPPD) 
using  Integrated  Product  Teams  (IPT).  This  management  philosophy  emphasizes  the 
potential  of  collective  knowledge  via  small  organizations  with  cross-functional  or  multi¬ 
disciplinary  members  (OUSD,  1998).  Interestingly,  the  ideas  in  this  IPPD/IPT 
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philosophy  of  work  implementation  and  problem  solving  are  also  embodied  and 
magnified  in  emerging  thought  regarding  command  and  control  (C2)  in  tactical  military 
organizations.  The  text  Power  to  the  Edge  recognizes  the  benefit  of  using  information- 
age  technology  to  transfer  knowledge  and  power  to  the  point  of  an  organization’s 
interaction  with  its  environment  (Alberts  &  Hayes,  2003). 


INTEGRATED  PRODUCT  TEAMS 


IPT  =  Integrated  Product  Team 
PIT  =  Program  Integration  Team 


Figure  3.  Example  (Aircraft)  PMO  IPT  Structure 

(adapted  from  DAU,  2004) 


Another  concept  pertinent  to  our  introduction  is  that  of  work  and  organizational 
hierarchy.  Nobel  Prize  winner  Herbert  Simon  argues,  from  his  observation  of 
complexity  in  things  both  natural  and  artificial,  that  complex  systems  evolve  from  simple 
systems.  And  they  do  so  more  rapidly  when  there  are  stable,  intermediate  forms  or  sub¬ 
systems  (like  modules  or  “units  of  action”).  Moreover,  he  argues  the  resulting  evolution 
into  the  complex  system  will  be  hierarchic,  including  systems  such  as  organizations 
(Simon,  1981).  But  an  important  observation  is  also  made  by  Koestler,  who  studies 
hierarchies  in  social  organizations.  He  notes  that  sub-systems  exist  only  as  entities 
relative  to  their  positions  in  the  hierarchy.  He  proposes  the  word  “holon”  to  describe  the 
hybrid  nature  of  individual  organizations  within  larger  organizations/systems.  Holons  are 
unique  and  self-contained  wholes  to  their  subordinated  parts.  But  at  the  same  time,  they 
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are  also  dependent  parts  of  the  larger  hierarchy  (or  “holarchy,”  as  Koestler  termed 
structures  consisting  of  them).  He  views  holons  as  autonomous,  self-reliant  units  which 
have  their  own  independence,  and  which  cope  with  contingencies  without  asking  higher 
authorities  for  instructions.  Yet,  they  remain  subordinate  ultimately,  subject  to  control 
from  higher  authorities.  The  term  seems  somewhat  analogous  to  edge  in  the 
conceptualization  of  Edge  organizations  (Alberts  &  Hayes,  2003).  Such  concepts  of  unit 
knowledge,  empowerment  and  relative  autonomy  within  organizational  structures  are 
key  to  our  design  of  various  organizations  for  experimentation. 

Organization  Theory 

Classic  organization  theory  holds  that  organizational  structures  must  change  in 
response  to  contingencies  of  size,  technology,  environment  and  other  factors.  Indeed,  it 
is  accepted  widely  that,  when  faced  with  uncertainty  (a  situation  with  less  information 
than  is  needed),  the  appropriate  management  response  should  be  either  to  redesign  the 
organization  for  the  task  at  hand  or  to  improve  information  flows  and  processing 
(Galbraith,  1973).  Van  Creveld  (1985)  applies  this  same  principle  to  command  and 
control  of  combat  elements  in  war.  He  argues  that  the  command  structure  must  either 
create  a  greater  demand  for  information  (vertically,  horizontally,  or  both)  and  increase 
the  size  and  complexity  of  the  directing  organization,  or  it  must  enable  the  local  forces 
to  deal  semi-independently  with  the  situation.  His  central  theme  is  that  decentralized 
control  is  the  superior  method  of  dealing  with  uncertainty,  whether  with  the  task  at  hand 
or  with  transformation  of  the  organization  itself.  Research  by  Van  de  Ven  and  Delbecq 
(1986)  has  shown  further  that  as  complexity  and  uncertainty  increase,  hierarchical 
management  control  and  vertical  communication  strategies  are  considered  inferior  to 
less  formal  organizations  with  horizontal  communication  channels. 

Another  classical  concept  of  organizational  theory  is  Ashby’s  Law  of  Requisite 
Variety  (Ashby,  1960).  This  states  loosely  that,  in  order  to  cope  with  the  variety  of 
challenges  imposed  by  it,  the  internal  capabilities  of  a  system  must  be  as  diverse  as 
those  required  by  its  environment.  Organizational  evolution  and  survival  are  dependent 
upon  requisite  variety,  particularly  in  environmental  contexts  that  are  dynamic  and 
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unpredictable.  Ashby’s  Law  suggests,  too,  that  the  organization’s  structure  and  control 
strategy  must  be  matched  to  its  environment  to  enhance  performance.  Open  and 
flexible  management  styles  and  processes  are  required  often  for  dynamic  market  and 
technological  conditions.  Further,  research  by  Burrell  and  Morgan  (Morgan,  1997) 
indicates  that  any  incongruence  among  management  processes  and  the  organization’s 
environment  tend  to  reduce  organizational  effectiveness. 

What  the  cumulative  research  appears  to  support  is  that,  for  large,  complex 
hierarchies  such  as  the  Department  of  Defense — which  operate  in  today’s  environment 
of  program  complexity,  evolving  requirements,  and  rapidly  changing  technology — 
decentralized  control  and  empowerment  should  be  an  organizational  strength. 
Notwithstanding  such  cumulative  research,  however,  organizational  hierarchies  persist 
(Leavitt,  2004).  Indeed,  for  DoD  acquisition  in  particular,  the  command  structure  has 
remained  relatively  stable  since  the  late  1980s.  Although  the  current  command 
structure  is  arguably  flatter  and  more  streamlined  now  than  it  was  in  the  1970s  and 
before,  it  remains  fundamentally  hierarchical,  centralized  and  rule-driven.  Only  through 
the  major  reform  initiatives  of  the  1980s  and  1990s  did  the  acquisition  organization’s 
“chain  of  command”  become  as  streamlined  as  it  now  is  (Packard  Commission,  1986). 


Computational  Experimentation 

Drawing  heavily  from  Nissen  and  Buettner  (2004),  we  assert  that  throughout  the 
era  of  modern  science,  a  chasm  has  persisted  between  laboratory  and  field  research. 
On  one  side,  the  laboratory  provides  unparalleled  opportunity  for  controlled 
experimentation.  Through  experimentation,  the  researcher  can  manipulate  only  a  few 
variables  of  interest  at  a  time  and  can  minimize  the  confounding  associated  with  the 
myriad  factors  affecting  complex  systems  and  processes  in  the  field  (Box  et  al.,  1978; 
Johnson  &  Wichern,  1992).  However,  limitations  of  laboratory  experimentation  are 
known  well  (Campbell  &  Stanley,  1973)  and  are  particularly  severe  in  the  domain  of 
acquisition.  In  acquisition  experimentation,  such  limitations  center  on  problems  with 
external  validity.  Laboratory  conditions  can  seldom  replicate  the  complexity,  scope  and 
scale  of  the  physical  organizations  and  systems  of  interest  for  research.  Experiments 
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also  include  problems  with  generalizability.  Many  experiments  utilize  samples  of 
convenience  (esp.  university  students)  instead  of  working  professionals.  This  practice 
calls  into  question  how  closely  the  associated  experimental  results  are  representative  of 
acquisition  behavior  in  operational  organizations.  These  same  concerns  pertain  also  to 
analytical  methods  (e.g.,  mathematical  analysis,  optimization;  see  Chiang,  1984;  Lapin, 
1985).  Most  such  methods  use  theoretical  concepts  as  variables,  not  operationalized 
constructs.  And,  of  course,  analytical  models  do  not  involve  real  people,  systems  and 
organizations. 

On  the  other  side,  field  research  provides  unparalleled  opportunity  for  realism 
(Denzin  &,  Lincoln,  1994).  The  researcher  in  the  field  can  study  full-scale  artifacts  in 
operational  environments  (Yin,  1994)  and  can  minimize  the  abstraction  away  from 
working  people,  systems  and  organizations  (Glaser  &  Strauss,  1967).  However, 
limitations  of  field  research  are  also  known  well  (Campbell  &  Stanley,  1973)  and  are 
particularly  severe  in  the  acquisition  domain.  In  acquisition  field  research,  such 
limitations  center  on  problems  with  internal  validity.  Field  research  affords  little 
opportunity  for  controlled  experimentation  (cf.  Cook  &  Campbell,  1979).  Also, 
confounding  data  often  results  from  the  myriad  influences  on  complex  systems  and 
organizations  that  cannot  be  isolated  in  the  field.  This  diversity  makes  it  difficult  to 
identify  and  trace  the  causes  of  differential  behaviors — better  as  well  as  worse — in 
acquisition. 

As  implied  by  the  name,  computational  experiments  are  conducted  via  computer 
simulation.  As  such,  they  offer  all  of  the  cost  and  time  advantages  of  computational 
analysis.  But,  computational  experiments  go  beyond  most  simulations.  Rigorous 
experimental  designs  are  employed  to  capture  the  benefits  of  laboratory 
experimentation.  The  variables  affecting  physical  systems  and  organizations  in  the  field 
can  be  isolated  and  examined  under  controlled  conditions.  This  type  of  analysis  also 
addresses  the  internal  validity  and  confounding  limitations  of  field  research.  Yet, 
computational  experiments  can  be  conducted  at  a  fraction  of  the  cost  and  time  required 
to  set  up  and  run  experiments  with  human  subjects  in  the  laboratory.  Further,  through 
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external  validation,  computational  models  can  demonstrate  fidelity  emulation  of  the  key 
qualitative  and  quantitative  behaviors  of  the  physical  systems  and  organizations  they 
represent.  This  ability  addresses  the  problems  with  external  validity  and  generalizability 
noted  above. 

It  is  important  to  note:  computational  modeling  and  simulation  are  not  new 
techniques  for  the  study  of  acquisition.  For  instance,  a  major  DoD  initiative  called 
simulation-based  acquisition  has  sought  to  educate  the  workforce  about  modeling  and 
simulation  (DSMC,  1998).  And  DoD  policy  has  called  for  extensive  use  of  modeling  and 
simulation  techniques  in  program  planning  and  execution  (Gansler,  1998).  But, 
simulation-based  acquisition  has  suffered  to  date  from  problems  with  internal  and 
external  validity  alike.  Such  problems  are  not  inherent  to  simulation  methods  or  tools 
per  se.  Rather,  they  stem  from  models  lacking  theoretically  rooted  behaviors,  externally 
validated  results,  and  experimental  controls.  Our  approach  to  computational 
experimentation  obviates  such  problems  deliberately. 

Figure  4  illustrates  the  essential  elements  of  computational  experimentation  as  a 
research  method.  The  top  of  the  figure  includes  a  shape  to  depict  the  bridge  metaphor 
associated  with  this  method,  as  it  spans  a  wide  gap  between  laboratory  and  field 
methods.  From  the  left  side  of  this  “bridge,”  two  arrows  represent  inputs  to  describe  the 
behaviors  of  computational  models.  Organization  theory,  which  is  predicated  upon 
many  thousands  of  studies  over  the  last  half  century,  provides  the  basis  for  most  such 
behaviors.  Behaviors  pertaining  to  organizational  factors  such  as  centralization,  division 
of  labor,  task  interdependence,  function,  coordination,  formalization,  technology  and 
information  processing  from  organization  theory  are  captured  well.  Where  extant  theory 
does  not  address  well  a  behavior  of  interest  (e.g.,  knowledge  flows),  ethnographic  and 
like  immersive  field  studies  (Bernard,  1998)  are  conducted  to  understand  the  associated 
organizational  behaviors.  Because  organization  theory  is  general  and  not  based  on  any 
single  organization,  the  associated  behaviors  have  broad  applicability  across 
organizations  in  practice.  This  provides  for  the  generalizability  attainable  through  the 
method  of  computational  experimentation. 
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Figure  4.  Bridge  Method 

(adapted  from  Nissen  &  Buettner,  2004) 

From  the  bottom  of  the  “bridge,”  an  arrow  represents  the  use  of  computer  models 
to  represent  organizations  and  to  emulate  their  key  behaviors.  Some  variety  exists  in 
terms  of  specific  implementations,  but  most  computer  models  adhere  to  standards, 
norms  and  conventions  associated  with  the  COT  field.  The  central  goal  is  to  develop 
computer  models  that  emulate  the  key  behaviors  of  organizations  and  to  use  such 
models  to  examine  alternate  methods  of  organization  and  coordination.  As  such,  COT 
shares  with  acquisition  a  focus  on  many  factors  of  importance. 

From  the  right  side  of  the  “bridge”  in  the  figure,  one  arrow  represents  a 
requirement  in  our  approach  for  model  validation.  Through  validation,  the  organizational 
behaviors  emulated  by  computer  models  are  examined  and  compared  with  those  of 
operational  organizations  in  the  field.  We  view  this  comparison  as  essential,  for  it 
provides  confidence  that  the  behaviors  emulated  by  the  computer  model  have  sufficient 
fidelity  to  mirror  faithfully  the  behaviors  of  the  operational  organizations  they  represent. 
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This  provides  for  the  external  validity  attainable  through  computational  experimentation. 
It  is  important  to  note,  not  all  COT  models  are  subjected  to  such  validation.  Many 
researchers  use  computational  models  to  conduct  theorem-proving  studies,  which  are 
valuable  in  their  own  right  to  demonstrate  various  aspects  of  organization  theory.  But 
without  validation,  researchers  have  difficulty  making  claims  that  such  theory  mirrors  the 
behavior  of  organizations  in  the  field.  Hence,  validation  represents  an  important 
characteristic  of  distinguishing  computational  experimentation  (as  the  research  method 
described  specifically  in  this  article)  from  COT  in  general. 

Finally,  from  the  top  of  the  “bridge,”  an  arrow  represents  the  use  of  experimental 
controls  in  research.  Following  the  same,  rich  set  of  experimental  designs  available  to 
laboratory  researchers  (e.g.,  full-factorial,  Latin  Squares,  blocking  with  replication), 
computational  experimentation  as  a  research  method  can  be  used  to  control  myriad 
factors  and  to  manipulate  just  one  or  a  few  variables  at  a  time  (e.g.,  searching  for 
causality  relations).  Further,  the  same  experimental  design  and  setup  can  be  replicated 
any  number  of  times — for  instance,  using  Monte  Carlo  techniques  or  other 
computational  approaches  to  introduce  variation.  This  provides  for  the  internal  validity 
attainable  through  computational  experimentation.  The  combination  of  these  “bridge” 
inputs — organization  theory  and  ethnography,  computer  models,  validation  and 
control — allows  the  method  of  computational  experimentation  to  be  understood  in  terms 
of,  and  to  indeed  inherit,  the  various  properties  of  its  constituent  elements. 

Figure  4  illustrates  also  the  bridging  nature  of  computational  experimentation  as 
a  research  method.  On  the  left  side,  we  depict  analytical  and  laboratory  methods  and 
we  summarize  their  key  advantages  (e.g.,  low-cost  &  fast  studies,  good  experimental 
control  &  internal  validity)  and  disadvantages  (e.g.,  poor  external  validity  & 
generalizability).  On  the  right  side,  we  depict  field  methods  in  similar  fashion  to 
summarize  their  key  advantages  (e.g.,  good  external  validity  &  generalizability)  and 
disadvantages  (e.g.,  high  cost  &  time  consuming,  poor  experimental  control  &  internal 
validity).  Notice  from  their  relative  advantages  and  disadvantages  how  the  two  classes 
of  research  methods  complement  one  another.  Field  methods  are  strong  in  the  areas 
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where  analytical  and  laboratory  methods  are  weak,  and  vice  versa.  As  an  alternate 
research  method,  computational  experimentation  mitigates  weakness  of  both  classes. 
For  instance,  it  enables  good  experimental  control  and  internal  validity  as  in  laboratory 
methods.  It  also  promotes  good  generalizability  and  external  validity  as  in  field  methods. 

Nonetheless,  every  research  method  is  flawed  in  some  respects.  In  our  present 
case,  when  used  in  isolation,  computational  experimentation  is  not  as  good  as  either 
method  at  its  best.  For  instance,  because  computational  experimentation  uses 
computer  models  of  people  in  organizations  instead  of  real  people,  it  is  weaker  in  this 
respect  than  laboratory  experimentation  is.  This  same  use  of  computer  models  instead 
of  real  people  also  makes  computational  experimentation  weaker  than  field  methods 
are.  This  is  why  we  describe  computational  experimentation  as  a  bridge  method :  it 
bridges  the  chasm  between  experimental  and  field  research  methods;  yet,  it  serves  best 
to  complement,  not  to  replace,  such  methods. 


ACQUISITION  RESEARCH 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 
NAVAL  POSTGRADUATE  SCHOOL 


-  17- 


Research  Design 


This  discussion  of  the  research  design  is  organized  into  three  parts:  1)  agent- 
based  modeling  environment,  2)  computational  acquisition  organization  model,  and  3) 
experimental  design. 

Agent-Based  Modeling  Environment 

In  this  section,  we  build  upon  current  advances  in  VDT  research  to  describe  the 
agent-based  modeling  environment  used  here  for  computational  experimentation. 
Drawing  heavily  from  Nissen  and  Levitt  (2004),  we  first  summarize  the  stream  of 
research  associated  with  VDT  and  then  describe  its  modeling  environment. 

Virtual  Design  Team  Research 

The  VDT  Research  Program  (VDT,  2004)  reflects  the  planned  accumulation  of 
collaborative  research  over  two  decades  to  develop  rich  theory-based  models  of 
organizational  processes.  Using  an  agent-based  representation  (Cohen,  1992;  Kunz  et 
al. ,  1998),  micro-level  organizational  behaviors  have  been  researched  and  formalized  to 
reflect  well-accepted  organization  theory  (Levitt  et  al.,  1999).  Extensive  empirical 
validation  projects  (e.g.,  Christiansen,  1993;  Thomsen,  1998)  have  demonstrated 
representational  fidelity  and  have  shown  how  the  emulated  behaviors  of  VDT 
computational  models  correspond  closely  with  a  diversity  of  enterprise  processes  in 
practice. 

The  VDT  research  program  continues  with  the  goal  of  developing  new  micro¬ 
organization  theory  and  of  embedding  it  in  software  tools  that  can  be  used  to  design 
organizations  in  the  same  way  that  engineers  design  bridges,  semiconductors  or 
airplanes:  through  computational  modeling,  analysis  and  evaluation  of  multiple, 
alternate  prototype  systems.  Clearly,  this  represents  a  significant  challenge  in  the 
domain  of  organizations.  Micro-theory  and  analysis  tools  for  designing  bridges  and 
airplanes  rest  on  well-understood  principles  of  physics  (e.g.,  involving  continuous 
numerical  variables,  describing  materials  whose  properties  are  relatively  easy  to 
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measure  and  calibrate),  and  analysis  of  such  physical  systems  yields  easily  to 
differential  equations  and  precise  numerical  computing. 

In  contrast,  theories  describing  the  behavior  of  organizations  are  characterized 
by  nominal  and  ordinal  variables,  with  poor  measurement  reproducibility,  and  verbal 
descriptions  reflecting  significant  ambiguity.  Unlike  the  mathematically  representable 
and  analyzable  micro-behaviors  of  physical  systems,  the  dynamics  of  organizations  are 
influenced  by  a  variety  of  social,  technical  and  cultural  factors,  are  difficult  to  verify 
experimentally,  and  are  not  as  amenable  to  numerical  representation,  mathematical 
analysis  or  precise  measurement.  Moreover,  quite  distinct  from  physical  systems, 
people  and  social  interactions — not  molecules  and  physical  forces — drive  the  behavior 
of  organizations.  Hence,  such  behaviors  are  fundamentally  non-deterministic  and 
difficult  to  predict  at  the  individual  level.  Thus,  people,  organizations  and  business 
processes  are  qualitatively  different  from  bridges,  semiconductors  and  airplanes.  And  it 
is  irrational  to  expect  the  former  to  ever  be  as  understandable,  analyzable  or  predictable 
as  the  latter.  This  represents  a  fundamental  limitation  of  the  approach. 

Within  the  constraints  of  this  limitation,  however,  we  can  still  take  great  strides 
beyond  relying  upon  informal  and  ambiguous,  natural-language  textual  description  of 
organizational  behavior  (e.g.,  the  bulk  of  extant  theory).  For  instance,  the  domain  of 
organization  theory  is  imbued  with  a  rich,  time-tested  collection  of  micro-theories  that 
lend  themselves  to  qualitative  representation  and  analysis.  Examples  include 
Galbraith's  (1977)  information-processing  abstraction,  March  and  Simon’s  (1958) 
bounded  rationality  assumption,  and  Thompson’s  (1967)  task-interdependence 
contingencies.  Drawing  from  this  theory  base,  we  employ  symbolic  (i.e.,  non-numeric) 
representation  and  reasoning  techniques  from  established  research  on  artificial 
intelligence  to  develop  computational  models  of  theoretical  phenomena.  Once 
formalized  through  a  computational  model,  the  symbolic  representation  is  “executable,” 
meaning  it  can  emulate  the  dynamics  of  organizational  behaviors. 

Even  though  the  representation  is  qualitative  (e.g.,  lacking  the  precision  offered 
by  numerical  models),  through  commitment  to  computational  modeling,  it  becomes 
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semi-formal  (e.g.,  different  people  viewing  the  model  can  agree  on  what  it  describes), 
reliable  (e.g.,  the  same  sets  of  organizational  conditions  and  environmental  factors 
generate  the  same  sets  of  behaviors),  and  explicit  (e.g.,  much  ambiguity  inherent  in 
natural  language  is  obviated).  Particularly  when  used  in  conjunction  with  the  descriptive 
natural  language  theory  of  our  extant  literature,  this  represents  a  substantial  advance. 
Further,  once  a  model  has  been  validated  to  emulate  accurately  the  qualitative 
behaviors  of  the  field  organization  it  represents,  it  can  be  used  to  examine  a  multitude  of 
cases  (e.g.,  many  more  and  diverse  than  observable  in  practice)  under  controlled 
conditions  (e.g.,  repeating  the  same  events  multiple  times,  manipulating  only  one  or  a 
few  variables  at  a  time  through  repeated  trials,  stopping  the  action  for  interpretation). 
These  features  alone  offer  great  promise  in  terms  of  theory  development  and  testing. 

Additionally,  although  organizations  are  inherently  less  understandable, 
analyzable  and  predictable  than  physical  systems  are,  and  the  behavior  of  people  is 
non-deterministic  and  difficult  to  model  at  the  individual  level,  it  is  known  well  that 
individual  differences  tend  to  average  out  when  aggregated  cross-sectionally  and/or 
longitudinally.  Thus,  when  modeling  aggregations  of  people  in  the  organizational 
context  (e.g.,  work  groups,  departments,  firms),  one  can  augment,  with  certain  aspects 
of  numerical  representation,  the  kind  of  symbolic  model  from  above.  For  instance,  the 
distribution  of  skill  levels  in  an  organization  can  be  approximated — in  aggregate — by  a 
Bell  Curve;  the  probability  of  a  given  task  incurring  exceptions  and  requiring  rework  can 
be  specified — organization  wide — by  a  distribution;  and  the  unpredictable  attention  of  a 
worker  to  any  particular  activity  or  event  (e.g.,  new  work  task,  communication,  request 
for  assistance)  can  be  modeled — stochastically — to  approximate  collective  behavior.  As 
another  instance,  specific  organizational  behaviors  can  be  simulated  hundreds  of 
times — such  as  through  Monte  Carlo  techniques — to  gain  insight  into  which  results  are 
common  and  expected  versus  those  that  are  rare  and  exceptional. 

Of  course,  applying  numerical  simulation  techniques  to  organizations  is  nothing 
new  (e.g.,  see  Law  &  Kelton,  1991).  But  this  approach  enables  us  to  integrate  the  kinds 
of  dynamic,  qualitative  behaviors  emulated  by  symbolic  models  with  quantitative 
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aggregate  dynamics  generated  through  discrete-event  simulation.  It  is  through  such 
integration  of  qualitative  and  quantitative  models — bolstered  by  strong  reliance  upon 
well-established  theory  and  commitment  to  empirical  validation — that  our  approach 
diverges  most  from  extant  research  methods  and  offers  new  insight  into  the  dynamics  of 
organizational  behavior. 

VDT  Modeling  Environment 

Here  we  provide  a  brief  overview  of  the  VDT  modeling  environment.  The 
development  and  evolution  of  VDT  has  been  described  in  considerable  detail  elsewhere 
(e.g.,  Cohen,  1992;  Christiansen,  1993;  Jin  &  Levitt,  1996;  Thomsen,  1998;  Kunz  et  al. , 
1998;  Levitt  et  al.,  1999;  Nogueira,  2000;  VDT,  2004),  so  we  do  not  repeat  such 
discussion  here.  The  VDT  modeling  environment  has  been  developed  directly  from 
Galbraith’s  information-processing  view  of  organizations.  This  information-processing 
view  has  two  key  implications  (Jin  &  Levitt,  1996).  The  first  is  ontological:  we  model 
knowledge  work  through  interactions  of  tasks  to  be  performed,  actors  communicating 
with  one  another  and  performing  tasks,  and  an  organization  structure  that  defines 
actors’  roles  and  that  constrains  their  behaviors.  In  essence,  this  amounts  to  overlaying 
the  task  structure  on  the  organization  structure  and  to  developing  computational  agents 
with  various  capabilities  to  emulate  the  behaviors  of  organizational  actors  performing 
work. 


Figure  5  illustrates  this  view  of  tasks,  actors  and  organization  structure.  As 
suggested  by  the  figure,  we  model  the  organization  structure  as  a  network  of  reporting 
relations  which  can  capture  micro-behaviors  such  as  managerial  attention,  span  of 
control  and  empowerment.  We  represent  the  task  structure  as  a  separate  network  of 
activities,  which  can  capture  organizational  attributes  such  as  expected  duration, 
complexity  and  required  skills.  Within  the  organization  structure,  we  further  model 
various  roles  (e.g.,  marketing  analyst,  design  engineer,  manager),  which  can  capture 
organizational  attributes  such  as  skills  possessed,  level  of  experience  and  task 
familiarity.  Within  the  task  structure,  we  further  model  various  sequencing  constraints, 
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interdependencies  and  quality/rework  loops — which  can  capture  considerable  variety  in 
terms  of  how  knowledge  work  is  organized  and  performed. 

As  suggested  also  by  the  figure,  each  actor  within  the  intertwined  organization 
and  task  structures  has  a  queue  of  information  tasks  to  be  performed  (e.g.,  assigned 
work  activities,  messages  from  other  actors,  meetings  to  attend)  and  a  queue  of 
information  outputs  (e.g.,  completed  work  products,  communications  to  other  actors, 
requests  for  assistance).  Each  actor  also  processes  such  tasks  according  to  how  well 
the  actor’s  skill  set  matches  those  required  for  a  given  activity,  the  relative  priority  of  the 
task,  the  actor’s  work  backlog  (i.e. ,  queue  length),  and  how  many  interruptions  divert  the 
actor’s  attention  from  the  task  at  hand.  Collective  task  performance  is  constrained 
further  by  the  number  of  individual  actors  assigned  to  each  task,  the  magnitude  of  the 
task,  and  both  scheduled  (e.g.,  work  breaks,  ends  of  shifts,  weekends  and  holidays) 
and  unscheduled  (e.g.,  awaiting  managerial  decisions,  awaiting  work  or  information 
inputs  from  others,  performing  rework)  downtime. 


Figure  5.  VDT  Information  Processing  View  of  Knowledge  Work 

(adapted  from  Nissen  &  Levitt,  2004) 
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The  second  implication  is  computational:  both  primary  work  (e.g.,  planning, 
design,  management)  and  coordination  work  (e.g.,  group  tasks,  meetings,  joint  problem 
solving)  are  modeled  in  terms  of  work  volume.  This  construct  is  used  to  represent  a  unit 
of  work  (e.g.,  associated  with  a  task,  a  meeting,  a  communication)  within  the  task 
structure.  In  addition  to  symbolic  execution  of  VDT  models  (e.g.,  qualitatively  assessing 
skill  mismatches,  task-concurrency  difficulties,  decentralization  effects)  through  micro¬ 
behaviors  derived  from  organization  theory,  the  discrete-event  simulation  engine 
enables  (virtual)  process  performance  to  be  assessed  (e.g.,  quantitatively  projecting 
task  duration,  cost,  rework,  process  quality). 

Clearly,  quantitative  simulation  places  additional  burden  on  the  modeler  in  terms 
of  validating  the  representation  of  a  knowledge-work  process,  which  generally  requires 
fieldwork  to  study  an  organization  in  action.  The  VDT  modeling  environment  benefits 
from  extensive  fieldwork  in  many  diverse  enterprise  domains  (e.g.,  power-plant 
construction  and  offshore  drilling,  see  Christiansen,  1993;  aerospace,  see  Thomsen, 
1998;  software  development,  see  Nogueira,  2000;  healthcare,  see  Cheng  &  Levitt, 
2001;  others).  Through  the  process  of  “backcasting” — predicting  known  organizational 
outcomes  using  only  information  that  was  available  at  the  beginning  of  a  project — VDT 
models  of  operational  enterprises  in  practice  have  demonstrated  dozens  of  times  that 
emulated  organizational  behaviors  and  results  correspond  qualitatively  and 
quantitatively  to  their  actual  counterparts  in  the  field  (Kunz  et  al.,  1998). 

Viewing  VDT  as  a  validated  model  of  project-oriented  knowledge  work, 
researchers  have  begun  to  use  this  dynamic  modeling  environment  as  a  “virtual 
organizational  testbench”  to  explore  a  variety  of  organizational  questions,  such  as 
effects  of  distance  on  performance  (Wong  &  Burton,  2000)  or  to  replicate  classic 
empirical  findings  (Carroll  &  Burton,  2000).  Thus,  the  VDT  modeling  environment  has 
been  validated  repeatedly  and  longitudinally  as  representative  of  both  organization 
theory  and  enterprises  in  practice.  This  gives  us  considerable  confidence  in  its  results. 
Moreover,  VDT  is  designed  specifically  to  model  the  kinds  of  knowledge  work  and 
information-processing  tasks  that  comprise  the  bulk  of  acquisition  processes. 
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Computational  Acquisition  Organizational  Model 

In  our  experimental  efforts,  we  use  the  VDT  modeling  environment  to  represent 
work  associated  with  a  three-tier  acquisition  organization.  This  follows  our  discussion 
above,  and  it  is  representative  of  many  DoD  service-level  environments  today  (e.g., 
where  several  project  offices  report  into  one  program  executive  “Portfolio  Manager”  and 
then  up  to  a  Component  Acquisition  Executive,  and  often  into  yet  another  level  of 
decision-making).  VDT  is  capable  of  modeling  large,  complex,  operational 
organizations  in  great  detail;  it  has  been  demonstrated  repeatedly  to  emulate  well  the 
associated  behaviors  of  organizations  in  the  field.  But,  using  a  high-level  model  as  such 
helps  us  to  maintain  the  focus  of  this  expository  article  on  techniques  of  VDT  modeling 
and  computational  experimentation  (which  represents  one  of  our  primary  contributions), 
and  not  to  get  lost  in  the  details  of  the  organization  itself.  We  first  describe  the  VDT 
representation  and  then  illustrate  how  a  full-factorial  computational  experiment  can  be 
performed  upon  it. 

VDT  Acquisition  Model 

Figure  6  presents  a  screenshot  of  the  VDT  acquisition  program  model.  The 
model  is  comprised  of  five  developmental  system  acquisition  projects  (i.e.,  denoted  as 
lightly  colored  boxes).  Both  concurrent  and  sequential  projects/tasks  are  depicted  in  the 
model,  and  interdependencies  are  represented  among  them.  The  model  depicts  a 
simple  and  abbreviated  series  of  coordinated  research  and  development  efforts  which 
are  aligned  to  deliver  an  Advanced  Strike  capability  integrated  into  a  mobile  platform 
and,  subsequently,  are  enhanced  with  an  evolutionary  block  of  capability.  It  is  but  a 
representative  subset  of  what  could  be  a  larger,  more  complex,  and  more  detailed 
representation  of  such  a  program. 
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Figure  6.  VDT  Acquisition  Model  Screenshot 


The  coordination  links  (i.e.,  denoted  by  light  dashed  lines)  connecting  the 
coordinated  tasks  or  projects  denote  reciprocal  task  interdependencies  (Thompson, 
1967),  which  suggest  they  must  be  coordinated  closely  in  both  planning  and  execution. 
For  example,  integration  of  strike  capabilities  into  a  mobile  platform  requires 
coordination  among  engineers  for  interface  and  configuration  control  of  hardware, 
software,  and  other  factors.  VDT  emulates  the  added  coordination  effort  associated  with 
such  reciprocal  task  interdependencies.  The  rework  links  (i.e.,  denoted  by  dark  dashed 
lines)  connecting  tasks  from  different  mission  phases  denote  sequential  task 
dependencies,  which  suggest  the  predecessor  activities  must  be  accomplished 
effectively  in  order  for  the  successors  to  perform  well.  Strike  and  Mobility 
Enhancements,  for  instance,  depend  heavily  upon  success  of  the  Initial  Strike  and 
Mobility  Platform  efforts.  To  the  extent  that  such  predecessor  work  is  not  completed  or 
not  accomplished  effectively,  certain  aspects  may  have  to  be  reanalyzed  to  correct  any 
major  deficiencies. 


The  people  icons  depict  organizations  and  are  arranged  in  terms  of  the 
command-and-control,  or  decision-making,  hierarchy.  People  icons  represent  one  or 
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more  human  resources,  specified  in  Full  Time  Equivalents  (FTEs),  which  have  particular 
capabilities,  skill  levels  and  roles.  Where  a  skilled  actor’s  capability  matches  that 
required  for  an  acquisition  task,  the  resource  is  likely  to  perform  it  competently  and 
within  the  time  required.  If  the  actor  has  greater  or  lesser  skill,  the  time  required  to 
perform  the  task  can  be  appreciably  shorter  or  longer,  and  the  competency  of 
performance  can  be  notably  better  or  worse,  respectively.  Where  an  actor  does  not 
possess  the  required  capability  at  all,  the  task  will  be  in  jeopardy.  Such  relationships  are 
appealing  intuitively  and  reflect  well  many  organizational  behaviors. 

A  Senior  Executive  actor  sits  atop  the  acquisition  organization  model  and  has  a 
Portfolio  Manager  reporting  to  it.  Reporting  to  the  Portfolio  Manager  are  five  individual 
PMOs  (01  through  05)  with  different  roles  and  capabilities  within  them.  For  example, 
the  icon  labeled  “PM  02”  is  responsible  for  the  technological  enhancement  of  the  initial 
Strike  Capability.  Notice  the  VDT  representation  includes  a  work  task  structure  and  an 
organization  structure.  The  assignment  links  (i.e. ,  delineated  by  solid  lines)  denote 
which  organizational  actors  are  responsible  for  the  various  work  tasks.  Finally,  a  dark 
trapezoid  box  is  used  to  depict  recurring  meetings  (e.g.,  coordination  meetings, 
technical  reviews,  milestone  reviews)  that  must  be  attended  by  the  actors  connected  by 
links.  Meetings  consume  actors’  resources,  but  they  also  contribute  toward 
coordination. 

All  of  the  structural  elements  (e.g.,  work  tasks,  requirements  and 
interdependencies,  actor  capabilities,  skill  levels  and  roles,  organization  structure,  task 
structure  and  meeting  requirements)  of  this  VDT  model  are  developed  by  the  authors. 
Such  structural  elements  would  clearly  be  different  for  each  unique  organization  and 
process  model.  VDT  also  includes  several  dozen  environmental  variables  with  “normal” 
values  determined  empirically  by  prior  field  research.  These  include  factors  such  as  the 
level  of  uncertainty  and  noise  associated  with  a  project,  the  inherent  propensity  of  an 
organization  to  make  errors,  and  relative  concern  for  performance  quality  associated 
with  actors  at  different  levels  of  organizational  hierarchy.  These  and  other 
environmental  variables  can  be  changed  where  appropriate  to  reflect  a  wide  variety  of 
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different  organizations  and  contexts.  Other  factors  can  be  changed  to  reflect  different 
organizational  designs.  For  instance,  the  level  of  centralization  and  formalization  can  be 
varied  by  changing  design  variables.  The  corresponding  VDT  model  behaviors  have 
been  developed  empirically.  We  capitalize  upon  such  empirically  developed  behaviors 
to  design  and  compare  new  acquisition  organization  models  and  subject  them  to 
changing  environments. 

VDT  also  includes  several  performance  variables  for  comparison.  In  addition  to 
standard  simulation  measures  such  as  project  duration  and  cost,  VDT  also  includes 
measures  such  as  levels  of  rework,  coordination  and  delay,  in  addition  to  risk  measures 
keyed  to  various  attributes  of  importance  (e.g.,  tasks  left  undone,  missed 
communications,  project-level  errors).  Some  of  these  performance  variables  are 
correlated  often  with  one  another,  whereas  others  highlight  tradeoffs  that  must  be 
made.  In  other  words,  where  a  project  is  running  behind  schedule  but  on  budget,  a 
leader  or  manager  can  decide  to  employ  more  resources.  This  often  has  the  effect  of 
increasing  the  rate  of  progress  while  also  increasing  the  rate  of  expenditure.  Other 
tradeoffs  such  as  those  between  cost  and  risk  or  schedule  and  coordination  require 
balance  in  a  similar  fashion.  It  is  important  to  note  again  the  extensive  and  longitudinal 
validation  of  VDT  provides  considerable  confidence  that  the  organizational  behaviors 
emulated  by  our  computational  model  will  reflect  well  those  of  operational  organizations 
in  the  field. 


Experimental  Design 

As  is  appropriate  for  the  cumulative  accretion  of  knowledge  through  research, 
this  study  builds  upon  prior  work  using  VDT  methods  and  tools  to  examine  alternate 
organizational  designs  and  environmental  conditions.  For  instance,  Kim  and  Burton 
(2002)  use  VDT  to  model  projects  with  varying  levels  of  task  uncertainty  and 
centralization,  measuring  the  effects  on  cost,  schedule  and  risk  as  dependent 
performance  variables.  They  find  a  relationship  between  organizational  structure  and 
performance.  And  they  examine  project  risk,  measuring  the  likelihood  that  outputs  from 
a  project  will  not  be  integrated  at  completion,  or  that  the  integration  will  have  defects. 
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The  study  calls  attention  to  the  impact  of  centralized  control  on  organizational 
performance  in  light  of  task  uncertainty.  It  also  suggests  that  managers  should  pay 
attention  to  such  aspects  of  organizational  structure  and  should  consider  the  importance 
of  project  quality  in  addition  to  profitability  alone.  In  another  instance,  Nissen  and 
Buettner  (2004)  use  VDT  to  model  command  and  control  in  military  missions.  They 
model  organizations  having  varying  levels  of  bureaucracy,  coordination  and  knowledge, 
measuring  the  effects  on  mission  duration  and  risk  as  dependent  performance 
variables.  They  find  a  similar  relationship  between  organizational  structure  and  task 
performance  and  overall  risk,  and  they  suggest  that  organizational  leaders  must  choose 
and  balance  the  performance  measures  that  are  most  relevant  to  the  project’s 
environment  and  desired  outcomes. 

In  this  study,  we  emulate  the  behaviors  of  three  different  modeled  organizations 
which  vary  in  degrees  of  hierarchy,  centralization  and  formalization,  and  which  are 
subjected  to  different  levels  of  environmental  stress.  Briefly,  our  three  designs  of 
organizations  have  the  same  amount  of  work  volume  to  perform,  with  the  same  level  of 
team  experience  and  individual  skills  involved.  What  differs  among  them  is  their  degree 
of  autonomy  and  empowerment,  specified  by  several  VDT  constructs.  Therefore,  we 
build  upon  the  kinds  of  prior  research  noted  above,  and  we  extend  such  prior  research 
to  address  the  acquisition  domain.  We  also  extend  prior  research  through  the  greater 
number  and  variety  of  organizational  design  changes  and  degrees  of  environmental 
stress  examined  in  this  study. 

Figure  7  reflects  today’s  acquisition  organization  (labeled  “Typical”)  with  high 
centralization,  formalization  and  three  layers  of  decision  hierarchy,  somewhat  like  an 
ACAT  II  program  or  set  of  projects  within  the  DoD. 
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Figure  7.  Typical  Acquisition  Organization  Design  and  Project  Work 

In  contrast  to  the  typical  organization,  Figures  8  and  9  depict  two  alternate 
organizations  with  fewer  layers  of  decision  hierarchy  and  lower  centralization  and 
formalization.  The  first  organization  (labeled  “Decentralized”)  has  less  hierarchy  and 
control  overhead  in  its  management  structure.  Note  the  removal  of  the  “Senior 
Executive”  position  in  the  representation,  whose  VDT  role  of  PM  has  been  delegated 
lower  to  the  Portfolio  Manager,  now  labeled  as  “Leader.”  As  in  reality,  the  supervision 
structure  in  the  VDT  model  is  an  exception-handling  hierarchy.  It  is  the  chain  of 
command  for  information  and  decision  about  problems  discovered  in  the  course  of  a 
project.  Positions  of  PM  01 , 02,  and  others  still  act  within  the  VDT  simulation  as 
Subteam  Leaders  who  handle  some  exceptions  and  pass  others  up  the  hierarchy  for 
resolution. 
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The  second  organization,  depicted  in  Figure  9  (labeled  “Holonistic”)  has  no 
overhead  management  structure  at  all.  Here,  each  PM  position  in  the  figure  remains 
designated  a  Subteam  Leader  within  the  VDT  tool.  The  various  PMs  communicate  with 
one  another  directly.  Table  1  shows  the  VDT  settings  for  organizational  parameters  to 
be  tested.  Additional  modeling  detail  on  organizational  design  parameters  is  found  in 
Appendix  A. 
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Table  1.  Organizational  Design  Parameters 


Organizational  Parameter  Typical 

Decentralized 

Holonistic 

Centralization 

High 

Low 

Low 

Formalization 

High 

Low 

Low 

Matrix  Strength 

Low 

High 

High 

Hierarchy 

3  layers 

2  layers 

1  layers 

Sr-Cmd  (Sr  Exec  PM): 

1  FTE 

0 

0 

Mid-Cmd  (Port  Mgr  SL): 

1  FTE 

1 

0 

Operations  (PMOs): 

50  FTE 

50 

50 

Communication  Links 

0 

2 

5 

Info  Exchange  Prob 

0.1 

0.9 

0.9 

Application  Exp. 

Low 

Medium 

Medium 

Meetings 

More 

Less 

None 

Functional  Error  Prob 

0.1 

0.2 

0.2 

Project  Error  Prob 

0.1 

0.2 

0.2 

Rework  Links  Str 

30 

10 

10 

Team  Experience 

Medium 

Medium 

Medium 

Skill  Level/Matched 

Medium 

Medium 

Medium 

Environmental  stress  is  applied  to  our  three  organizational  designs  via  the  VDT 
constructs  requirement  complexity,  solution  complexity,  and  task  uncertainty 
(appropriate  to  environments  that  project  offices  often  face  with  technology  maturity, 
interoperability  requirements,  etc.)  as  well  as  higher  noise  (distractions)  and  increasing 
functional-  and  project-error  probabilities.  For  the  experiment,  each  of  these  three 
factors  is  specified  at  two  levels:  routine  and  stressed,  shown  in  Table  2  below.  Hence, 
a  full-factorial  design  consists  of  six  trials  (i.e.,  three  alternate  organizational  designs  x  2 
different  environmental  conditions),  which  we  designate  according  to  the  levels 
corresponding  to  a  set  of  environmental  factors.  Additional  modeling  detail  on 
environmental  parameters  is  presented  in  Appendix  B. 
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Table  2.  Environmental  Parameters 


Environmental  Parameter  Routine  Stressed 


Requirement  Complexity 

Medium 

High 

Solution  Complexity 

Medium 

High 

Uncertainty 

Medium 

High 

Noise 

0.3 

0.4 

Functional  Error  Probability 

0.1  &  0.2 

0.3  &  0.4 

Project  Error  Probability 

0.1  &  0.2 

0.3  &  0.4 

We  examine  the  dependent  variables  of  particular  interest  in  the  acquisition 
domain:  cost,  schedule  duration  and  project  risk.  We  also  make  note  of  the  maximum 
position  backlog,  rework  volume,  coordination  volume,  and  decision  wait  time,  as  these 
have  implications  for  managers  to  consider.  Schedule  is  important  to  project  managers, 
and  time  is  often  viewed  as  money  because  of  the  staff  that  must  be  paid  as  long  as 
they  are  retained — whether  productive  to  the  project  or  not.  Project  cost  is  measured  in 
$K,  and  pertains  to  staffing  costs  only,  as  no  material  costs  are  modeled  in  our 
experiment.  Project  risk,  as  mentioned  above,  is  represented  as  the  likelihood  of  an 
incomplete  project  outcome,  which  relates  directly  to  project  quality.  While  every  task 
within  a  project  may  not  be  critical  to  project  quality,  more  tasks  incomplete  or  defective 
place  the  overall  project  at  risk  for  failure.  Where  lives  are  at  stake,  such  as  in  new 
pharmaceutical  compounds,  new  passenger  aircraft,  or  defense  weapon  systems 
involving  lethality  and  survivability,  overall  project  risk  may  be  a  difficult  trade  for 
managers  also  concerned  with  project  cost  and  schedule. 
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Experimental  Results 


In  this  section,  we  report  on  the  results  of  our  computational  experiment. 
Summarized  in  Table  3  is  each  of  the  six  trials  in  this  full-factorial  experiment.  The  table 
includes  measures  for  project  cost,  schedule  and  risk,  in  addition  to  other  metrics  that 
can  provide  insight  into  organizational  dynamics  (rework  volume,  coordination  volume 
and  decision  wait). 


Table  3.  Experimental  Results 


Typical  Decentralized  Holonistic  Typical  Decentralized  Holonistic 
Organization  Organization  Organization  Organization  Organization  Organization 
Measure  in  Routine  in  Routine  in  Routine  Under  Stress  Under  Stress  Under  Stress 


Duration  (dys) 

556 

428 

407 

580 

604 

458 

Cost  $K 

$8,085 

$4,674 

54.565 

$8,561 

$6,708 

$4,973 

Project  Risk 

0.41 

0.54 

0.76 

0.37 

0.55 

0.76 

Max  Backlog  (dys) 

26 

12 

12 

30 

27 

19 

Work  Volume  (dys) 

4800 

4500 

4500 

4800 

4500 

4500 

Rework  Volume  (dys) 

124 

866 

465 

401 

2747 

740 

Coordination  Volume  (dys) 

3051 

423 

742 

3205 

952 

976 

Decision  Wait 

20 

54 

0 

67 

186 

0 

Examining  these  results,  we  see  that  the  baseline  organization — the  Typical 
Organization  in  Routine  environment — completes  the  series  of  projects  in  556  days,  at  a 
cost  of  $8,085  (K),  with  a  project  risk  index  of  0.41 .  While  these  are  the  three  primary 
success  measures  of  any  project,  the  VDT  simulation  provides  more  insight  in  terms  of 
position  backlog  (e.g.,  one  actor  got  26  days  behind  in  work  at  one  point  during  the 
project).  The  tool  can  also  identify  when  this  occurs  so  that  planners  can  split  tasks  or 
assign  more  resources  for  specific  tasks.  Work  volume  refers  to  the  amount  of  effort 
expected  to  complete  all  project  tasks  under  ideal  conditions  (e.g.,  no  noise,  errors  or 
miscommunications).  Rework  Volume  refers  to  the  simulated  time  needed  for  all 
positions  on  a  project  to  perform  required  rework.  Coordination  volume  is  the 
cumulative  time  positions  spend  during  a  project  processing  information  requests  from 
each  other,  attending  meetings,  and  other  coordinative  tasks.  Decision  Wait  measures 
the  cumulative  time  spent  by  positions  waiting  for  decisions  to  be  made  in  a  project. 
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These  values  for  our  baseline  case  provide  a  basis  for  comparison  with  results  for 
alternate  organizational  designs  and  environmental  conditions. 

Comparing  these  results  with  those  obtained  by  the  Decentralized  and  Holonistic 
Organizations  in  the  Routine  environment,  key  differences  are  apparent.  Decentralized 
and  Holonistic  organizations  fare  considerably  better  (in  terms  of  both  cost  and 
schedule)  in  the  routine  environment  than  their  more  Typical  counterpart  organizational 
design  does.  Program  schedule  or  duration  is  reduced  some  23%  (from  556  to  428 
days)  with  the  Decentralized  Organization  and  27%  (from  556  to  407  days)  by  changing 
organizational  structure  toward  a  more  Holonistic  design.  Program  cost  is  reduced 
similarly  by  42%  (from  $8085  (K)  to  $4674  (K))  with  Decentralized  and  44%  (from  $8085 
(K)  to  $4565  (K))  with  Holonistic  in  the  successive  design  iterations.  However,  project 
risk  increases  appreciably  in  both  alternate  organizations,  going  up  to  54%  and  then  to 
76%,  respectively,  in  Decentralized  and  Holonistic  designs.  Here,  we  find  that 
Decentralized  and  Holonistic  organizational  forms  offer  a  combination  of  advantages 
(e.g.,  shorter  schedule  duration,  lower  cost)  and  disadvantages  (e.g.,  higher  risk)  with 
respect  to  the  Typical  acquisition  organization  in  a  routine  environment. 

Upon  examination  of  these  organizational  designs  under  stress  environments, 
we  find  the  Typical  Organization  suffers  cost  and  schedule  growth  in  the  4-5%  range 
(i.e.,  580  days,  $8561  (K)),  with  a  slight  decline  in  project  risk  (0.37).  Decentralized  and 
Holonistic  organizations  under  stress  perform  better  in  the  cost  realm  with  22%  ($6708 
(K))  and  42%  ($4973  (K))  reductions  compared  with  the  Typical.  The  Decentralized 
design  reveals  longer  schedule  duration  (604  vs.  580  days),  but  the  Holonistic 
organization  shows  a  21%  decrease  (458  vs.  580  days).  Again,  project  risk  climbs  in 
stress  environments  to  55%  for  Decentralized  and  to  76%  for  the  Holonistic. 


Figure  10  illustrates  graphically  the  dynamic  relationship  we  find  between  cost 
and  organizational  design.  Notice,  in  the  routine  environment,  project  cost  decreases 
abruptly  with  a  shift  from  a  Typical  to  a  Decentralized  organizational  form.  But, 
negligible  additional  improvement  accrues  to  the  Holonistic  design.  Alternatively,  in  the 
stressed  environment,  the  Decentralized  organization  performs  better  than  the  Typical 
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does,  and  the  Holonistic  organization  performs  better  still.  Notice  also  how  costs  are 
higher  for  every  organizational  form  in  the  stressed  environment  than  they  are  in  the 
routine  one. 


Figure  10.  Relationship  between  Cost  and  Organizational  Design 

Figure  1 1  illustrates  the  relationship  between  risk  and  organizational  design. 
Here,  we  observe  a  monotonic  increase  in  risk  corresponding  to  progression  in 
organizational  form  from  Typical,  through  Decentralized,  to  Holonistic.  As  costs 
decrease  across  these  alternate  organizational  forms,  risk  increases  in  lock  step.  Unlike 
the  cost  results,  however,  the  stressed  environment  appears  to  exert  little  influence  in 
terms  of  risk. 
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Figure  11.  Relationship  between  Risk  and  Organizational  Design 

Interpreting  these  results,  the  researchers  found  that  the  less  centralized, 
formalized  and  hierarchical  organizational  designs  perform  better  in  terms  of  cost  and 
schedule  than  other  designs,  but  with  accompanying  project-quality  risk.  Interpreting 
these  results  further,  in  which  schedule  and  cost  are  of  primary  concern  to  all  project 
managers,  decentralized  control  (especially  in  stressed  environments)  may  provide  a 
more  cost-effective  approach.  Alternatively,  where  project  risk  or  quality  is  paramount, 
formalized  procedures,  vertical  information  flows,  and  centralized  decision-making 
typical  of  bureaucratic  organizational  forms  can  be  seen  as  superior.  This  reflects  a 
fundamental  tradeoff  between  performance  measures  and  organizational  design,  as 
conceptualized  generally  in  terms  of  Contingency  Theory  (Lawrence  &  Lorsch,  1967). 
And  as  Kim  and  Burton  (2002)  noted,  the  theory  is  actually  extended  by  the  evidence  of 
risk  coming  into  play  with  a  more  rapid  and  inexpensive  project  solution  afforded  by 
empowered  actors  with  relevant  information  at  their  organizational  edge. 

The  DoD,  like  sponsors  of  projects  in  the  FDA’s  pharmaceutical  arena  and  in  the 
FAA’s  commercial  aviation  arena,  is  averse  generally  to  risk  due  to  the  safety  and 
survivability  aspects  of  many  of  its  developmental  systems.  Indeed,  the  modeling  here 
can  be  viewed  as  confirmation  of  the  DoD’s  varying  levels  of  decision  hierarchy 
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correlating  to  estimated  program  dollar  thresholds  (stratification  of  acquisition  categories 
I  though  IV)  as  a  means  of  addressing  cost  risk.  However,  and  just  as  important  to 
illustrate,  high  levels  of  bureaucracy  place  considerable  stress  on  acquisition 
organizations  and  come  at  their  own  cost.  Are  40%  program  cost  growth  and  25% 
schedule  growth  commensurate  with  20  -  50%  program  risk  reduction?  Might  a 
commensurate  amount  of  risk  be  alleviated  through  less-expensive  means?  Clearly, 
tools  such  as  VDT  provide  a  new  way  of  gaining  insights  into  these  important  program 
considerations,  particularly  when  forming  organizations  for  the  management  of  weapon 
system  developments. 
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Conclusion 


The  DoD  is  a  large,  bureaucratic,  rule-intensive  organization  that  may  not  be 
suited  well  for  its  environment.  Building  upon  prior  research  on  acquisition  centralization 
and  knowledge  dynamics,  we  employ  computational  methods  to  assess  the  behavior 
and  performance  of  different  organizational  designs  in  varying  environments.  Our 
results  reinforce  Contingency  Theory  and  suggest  particular  characteristics  of  different 
acquisition  environments  that  make  one  form  relatively  more  or  less  appropriate  than 
another.  Practically,  answers  to  our  research  questions  have  direct  and  immediate 
application  to  acquisition  leaders  and  policy  makers.  Theoretically,  we  generalize  to 
broad  classes  of  organizations  and  prescribe  a  novel  set  of  organizational  design 
guides. 

In  this  study,  we  use  the  VDT  modeling  environment  to  represent  and  emulate 
the  behavior  of  an  acquisition  organization.  Although  the  Typical  acquisition 
organization  modeled  in  this  study  is  representative  of  such  organizations  in  practice, 
we  do  not  claim  to  have  experimented — even  computationally — with  an  operational 
organization.  Rather,  we  experiment  computationally  with  a  high-level  organizational 
model,  illustrating  the  method,  use  and  utility  of  our  approach  for  exposition.  We  then 
conceptualize  and  model  two  alternate  acquisition  organizations,  manipulating  key 
factors  of  their  organizational  designs.  We  subject  them  to  two  environmental  contexts, 
routine  and  stressed,  comparing  their  performance  in  terms  of  cost,  schedule  and  risk. 

In  routine  circumstances,  our  experimental  decentralized  and  holon-type 
organizations  out-perform  typical  hierarchies  in  measures  of  cost  and  schedule.  Under 
high  stress  from  task  uncertainty,  noise,  and  error  probability,  our  decentralized  and 
holon-type  organizations  completed  their  same  project  work  volume  as  well,  faster,  and 
for  less  cost  than  their  centralized  counterpart.  In  both  environments,  however,  our  less 
formal  organization  structures  yield  a  higher  project-quality  risk. 
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Our  findings  are  similar  to  those  of  other  VDT  researchers  who  find  the 
relationship  between  organizational  performance  improvements  and  increasing  project 
risk  from  decentralization  in  environments  of  uncertainty  (Kim  &  Burton,  2002)  and 
worker  knowledge  (Nissen  &  Buettner,  2004).  They  offer  an  extension  of  contingency 
theory  to  include  risk  as  a  dependent  variable  for  organizational  structures  and  project 
outcomes.  Our  results  reveal  the  same  relational  patterns  of  performance  capabilities 
among  the  three  organizational  designs  and  across  differing  stress  environments.  They 
underscore  complex  interactions  between  organizational  design  factors,  and  suggest 
fundamental  tension  and  decision  tradeoffs  between  important  performance  measures 
such  as  project  cost,  schedule  and  quality/risk. 

The  results  provide  several  implications  for  managerial  practices  and  application 
of  organization  theory  regarding  the  relationships  between  organizational  structure  and 
performance.  Understanding  when  the  bureaucracy  is  relatively  beneficial  and  how  this 
rigid  organizational  form  can  negatively  influence  project  cost  (and  positively  impact 
project  risk)  is  important  for  acquisition  practitioners  today.  The  apparent  implications 
are  that  adopting  a  decentralized  structure  in  accordance  with  contingency  theory  may 
not  lead  to  higher  unit  performance,  since  it  might  instead  produce  poorer  project 
quality.  But,  we  suspect  it  is  insufficient  to  only  assume  that  more  bureaucracy 
alleviates  risk  with  attendant  costs,  or  that  managers  must  simply  choose  either  fast  and 
cheap,  or  better  quality  results. 

In  the  early  1990’s,  with  a  goal  of  shortening  development  times,  reducing  cost, 
and  increasing  numbers  of  scientific  missions  flown,  NASA  adopted  a  “Faster,  Better, 
Cheaper”  approach  to  project  management.  This  management  philosophy  was 
implemented  in  spite  of  an  old  project  management  adage  that  project  managers  could 
have  any  two  of  these  performance  outcomes,  but  not  all  three  (Spear,  2000).  This 
maxim  is  supported  somewhat  by  the  findings  of  Lin  and  Carley  (1997)  regarding 
decision  accuracy  in  organizations  under  time  pressure.  After  the  several  unmanned 
mission  failures,  and  ultimately  the  February  2003  Columbia  disaster  that  claimed  the 
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lives  of  seven  astronauts,  an  analysis  of  NASA  failures  blamed  a  more  risk-tolerant 
culture  as  an  organizational  cause  of  the  accident. 

The  DoD,  having  large  complex  systems  with  inherent  risk  of  their  own,  is 
particularly  averse  to  risk  in  its  decision  structure  and  perhaps  even  its  organizational 
culture.  With  growing  federal  budget  deficits  and  base  re-alignments,  it  is  also 
particularly  cost-constrained.  And  with  accelerating  obsolescence  rates  of  weapon 
system  technology,  the  DoD  remains  under  considerable  pressure  to  reduce  project 
schedules  as  well.  Even  with  simple  models,  we  show  that  project  performance  can  be 
examined  with  various  organizational  designs  and  under  differing  environments. 

Perhaps  for  the  first  time — or  at  least  to  an  extent  unachievable  heretofore — we  show 
how  managers  can  gain  fundamental  insights  into  the  inherent  project  tradeoffs  in 
advance  of  making  project  decisions.  The  practical  significance  should  be  apparent 
immediately. 

These  experiments  support  propositions  that  information  processing  is  a  primary 
organizational  activity  and  is  associated  with  project  cost  and  duration  (i.e.  the  more 
information  processing  a  project  requires,  the  more  costly  and  lengthy  the  project 
becomes).  Certainly,  there  is  attendant  benefit  to  the  information  processed  as  well. 
However,  the  additional  measure  of  project-quality  risk  is  critical  for  many  types  of 
projects,  and  its  emerging  relationship  from  these  studies  and  our  most  recent  work 
begin  to  shape  a  new  hypothesis:  that  perhaps  there  is  an  optimal  organizational  design 
solution,  relative  to  cost,  duration  and  risk.  If  managers  can  ascertain  early  on  the 
criticality  (and  tolerable  level)  of  project-quality  risk,  they  can  perhaps  select  along  a 
continuum  the  level  of  organizational  hierarchy  and  centralization  needed  to  control 
project  outcomes.  Or,  reframing  the  question,  how  much  will  added  bureaucracy  cost  to 
alleviate  risk?  The  key  point  is:  the  answer  will  differ — necessarily — for  every  project.  A 
one-size-fits-all  acquisition  policy  is  naive  given  such  knowledge  and  our  ability  to 
emulate  organizational  performance  as  illustrated  in  this  article. 


Building  upon  the  VDT  constructs  introduced  in  this  article,  one  day  researchers 
may  even  develop  techniques  for  design  optimization  based  on  project  objectives  (e.g., 
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speed  vs.  risk)  and  environment.  Leaders,  managers  and  researchers  may  develop  the 
capability  to  design  organizations,  work  processes  and  technologies  using 
computational  techniques  comparable  to  those  employed  for  designing  airplanes, 
bridges  and  computers.  That  day  is  not  yet  here.  But,  through  research  along  these 
lines,  we  can  both  foresee  and  accelerate  its  arrival.  Meanwhile,  the  centralized  control 
that  dominates  current  acquisition  thinking  and  policy  merits  re-examination  in  light  of 
this  study.  Such  control  imposes  costs  as  well  as  accrues  benefits.  We  know  now  how 
to  measure  and  compare  them:  via  computational  experimentation. 
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Appendix  A:  Design  of  Organization  Models 


This  appendix  provides  additional  detail  about  the  design  of  our  experimental 
acquisition  organization  models.  Table  1  above  specifies  design  parameters  and  VDT 
simulation  settings  for  all  three  of  our  organizational  designs:  Typical,  Decentralized  and 
Holonistic.  We  reproduce  the  information  below  as  Table  4  and  discuss  the  various 
design  parameters. 


Table  4.  Organizational  Design  Parameters 


Organizational  Parameter  Typical 

Decentralized 

Holonistic 

Centralization 

High 

Low 

Low 

Formalization 

High 

Low 

Low 

Matrix  Strength 

Low 

High 

High 

Hierarchy 

3  layers 

2  layers 

1  layers 

Sr-Cmd  (Sr  Exec  PM): 

1  FTE 

0 

0 

Mid-Cmd  (Port  Mgr  SL): 

1  FTE 

1 

0 

Operations  (PMOs): 

50  FTE 

50 

50 

Communication  Links 

0 

2 

5 

Info  Exchange  Prob 

0.1 

0.9 

0.9 

Application  Exp. 

Low 

Medium 

Medium 

Meetings 

More 

Less 

None 

Functional  Error  Prob 

0.1 

0.2 

0.2 

Project  Error  Prob 

0.1 

0.2 

0.2 

Rework  Links  Str 

30 

10 

10 

Team  Experience 

Medium 

Medium 

Medium 

Skill  Level/Matched 

Medium 

Medium 

Medium 

Our  experiments  simulate  the  acquisition  efforts  of  five  small  project- 
management  offices  oriented  on  an  initial  strike  capability  (such  as  one  provided  by  a 
missile  or  direct-fire  weapon),  integration  into  a  mobile  platform  (such  as  a  ground  or  air 
vehicle),  and  followed  by  block  enhancements  to  both  sets  of  capability  from  insertion  of 
technology.  These  types  of  effort  are  common  within  program  executive  office  portfolios 
across  the  military  services.  To  simulate  the  completion  of  these  projects,  we  design 
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three  different  organizations  with  similar  resources,  but  varying  parameters  of 
organizational  control. 


Centralization,  Formalization  and  Matrix  Strength  comprise  a  group  of  variables 
that  work  together  in  our  modeling  tool  to  characterize  levels  of  bureaucratic 
organizational  control.  Low  Centralization  settings  in  the  Decentralized  and  Holonistic 
models  mean  decisions  are  made  by  the  individual  responsible  positions.  Centralization 
reflects  decision-making  in  the  organization — either  by  senior  project  positions  or  by 
“decentralized”  individuals.  There  is  more  communication  required  with  High 
Centralization.  Formalization  measures  how  formal  the  communication  is  in  an 
organization,  with  High  Formalization  meaning  that  communication  tends  to  occur  in 
formal  meetings  (despite  the  many  informal  communication  occurring  in  any  typical 
project  office),  and  Low  settings  reflecting  informal  communication  among  positions. 
Matrix  Strength  characterizes  the  probability  that  workers  will  attend  to  exchanges  of 
information — whether  via  meetings,  communications  about  tasks,  or  noise.  It  conveys 
“connectedness”  and  can  correspond  to  geographical  collocation  of  workers.  Typical 
major  program  acquisition  organizations  have  workers  and  decision  makers  distributed 
across  the  country,  and  there  is  a  greater  need  for  meetings.  The  Low  setting  for 
Typical  acquisition  organizations,  however,  reflects  high  meeting  quality  and 
complements  High  Formalization.  Conversely,  High  Matrix  Strength,  which 
complements  Low  Formalization,  characterizes  our  flatter,  more  Decentralized  or 
Holonistic  organizations.  In  the  “Typical”  model,  we  use  a  three-level  management 
hierarchy  with  two  different  full-time  equivalents  (FTE)  in  two  management  positions, 
acting  as  portfolio  manager  and  senior  executive.  We  reduce  to  two,  and  then  one  in 
the  other  derivations.  The  PM  and  SL  designations  beside  their  positions  are  VDT 
designations,  which  connote  decision-making.  The  50  FTEs  aside  the  Operations 
parameter  represent  project  management  office  personnel  in  five  distinct  project  work 
areas  that  are  interdependent,  and  become  more  strengthened  with  communication 
links,  used  as  such  by  the  VDT.  In  the  Typical  DoD  acquisition  organization,  we  have 
observed  it  is  common  for  individual  project  offices  within  a  PEO  to  communicate 
infrequently,  though  there  is  a  great  deal  of  vertical  communication  within  the  hierarchy 
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(as  evidenced  in  the  model  with  Meetings).  We  also  reduce  the  number  of  meetings  in 
the  successively  flatter  organizational  designs. 

Correspondingly,  hierarchical  communication  is  also  depicted  via  a  low  setting 
(0.1 )  of  Information  Exchange  Probability  for  the  Typical  Organization,  growing  much 
higher  (0.9)  in  the  flatter  designs.  This  is  characteristic  for  a  project  involving  mostly 
routine  daily  jobs  performed  by  skilled  workers  (Typical  design).  A  higher  value  is  given 
the  designs  with  more  highly  interdependent  tasks  that  are  being  performed  by  very 
busy  workers.  A  low  setting  of  Application  Experience  for  the  Typical  organization,  and 
set  at  the  entire  program  level,  describes  how  many  new  R&D  projects  the  positions 
may  have  worked  on  before,  in  spite  of  relative  individual  skill  levels  (which  are  all 
presumed  as  Medium  and  Matched  to  the  work  tasks  assigned  across  all  three 
designs).  The  Decentralized  and  Holonistic  organizations  are  envisioned  as  learning 
organizations,  with  the  benefit  of  some  Application  Experience  (set  at  Medium)  in  less 
complicated  information  processing  and  learning. 

Five  Rework  links  connect  all  tasks  in  the  Holonistic  design,  given  the  lack  of  an 
overhead  hierarchy  as  an  integrative  function.  This  is  opposed  to  two  links  in  the 
Decentralized  and  Typical  designs.  Likewise,  Rework  Strength  designations  shift  to 
reflect  success/failure  dependency  as  higher  in  the  Decentralized  and  Holonistic,  and 
lower  in  the  Typical  hierarchy,  commensurate  with  associated  task  interdependency.  In 
much  the  same  way,  the  stove-piped,  independent  efforts  within  the  Typical 
organization  hierarchy  are  represented  in  lower  Functional  and  Project  Error  Probability 
settings.  Higher  settings  for  the  flatter  organizational  designs  convey  the  challenge  of 
integration  and  alignment  they  must  face  without  an  overarching  control  entity. 

The  total  simulated  work  task  effort  of  all  organizational  designs  is  the  same, 
except  that  layers  of  management  in  the  Typical  configuration  have  their  own 
management  tasks  that  run  about  25%  of  the  duration  of  the  project’s  planned  timeline. 
The  Team  Experience  value  in  the  VDT  tool  affects  the  amount  of  information  exchange 
on  the  project  and  the  way  a  position’s  information  processing  speed  is  calculated. 
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Team  Experience  for  the  work  effort  is  set  to  Medium  for  all  organizations,  representing 
a  measure  of  how  successfully  the  team  has  performed  related  projects. 
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Appendix  B:  Environmental  Parameters 


This  appendix  provides  additional  detail  about  the  environmental  scenarios  that 
our  experimental  acquisition  organization  models  were  subjected  to.  Table  2  above 
specifies  environmental  parameters  and  VDT  simulation  settings  for  both  of  our 
environmental  conditions:  Routine  and  Stressed.  We  reproduce  the  information  below 
as  Table  5  and  discuss  the  various  parameters. 


Table  5.  Environmental  Parameters 


Environmental  Parameter 

Routine 

Stressed 

Requirement  Complexity 

Medium 

High 

Solution  Complexity 

Medium 

High 

Uncertainty 

Medium 

High 

Noise 

0.3 

0.4 

Functional  Error  Probability 

0.1  &  0.2 

0.3  &  0.4 

Project  Error  Probability 

0.1  &  0.2 

0.3  &  0.4 

Requirement  Complexity  describes  the  degree  of  task  complexity,  which  is 
relative  to  the  total  number  of  project  requirements  that  the  task  must  satisfy. 
Representative  of  current  DoD  acquisition  environments  are  state-of-the-art 
technologies  and  interoperability  requirements.  As  such,  there  is  a  common 
environment  of  at  least  a  Medium  setting  within  our  VDT  tool  for  organizations  under 
even  routine  circumstances.  An  even  more  highly  optimized  design  could  have  many 
tasks  with  a  High  Requirement  Complexity,  and  is  appropriate  for  our  “stressed” 
settings. 

Solution  Complexity  represents  the  number  of  solutions  to  which  a  task 
contributes.  The  degree  of  complexity  reflects  the  effect  a  task  has  on  the  tasks  that 
depend  on  it.  Thus,  for  routine  circumstances,  we  use  “Medium”  and  “High”  for  stressed 
scenarios. 
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Uncertainty  is  a  setting  regarding  the  amount  of  communication  across  links 
needed  for  a  task’s  (and  its  dependent  tasks’)  completion.  Task  Uncertainty  reflects  the 
effect  that  other  tasks  can  have  on  each  other  within  the  project.  Task  Coordination 
volume  and  the  number  of  communications  increase  with  higher  uncertainty,  so  we 
selected  “Medium”  for  routine  and  “High”  for  stressed  environments. 

Noise  Probability  describes  the  interruptions  or  distractions  that  detract  from  work 
on  project  tasks.  The  probability  of  noise  is  set  at  0.3  for  our  routine  scenarios  and  0.4 
for  stressed. 

Functional  Error  Probability  is  the  probability  that  a  task  will  fail  and  require 
rework.  Functional  errors  are  localized  to  an  individual  task  and,  thus,  only  cause 
rework  in  that  task.  Functional  errors  could  be  discovered  via  self-check,  a  project- 
review  meeting,  or  a  supervisory  review.  Depending  on  the  level  of  centralization  and 
hierarchy  in  the  project,  an  exception  can  be  handled  by  the  responsible  position  or 
someone  up  the  hierarchy.  When  a  functional  error  is  detected,  an  exception  is  sent  to 
the  responsible  position  or  to  a  supervisor,  generating  either  rework  of  the  task,  a  quick 
fix,  or  feigned  ignorance  of  the  problem.  Project  Error  Probability  is  the  probability  that 
a  task  will  fail  and  generate  rework  for  all  dependent  tasks  connected  to  it  by  rework 
links.  The  more  rework  links  there  are  in  a  project,  the  more  rework  is  generated  by  the 
exceptions  that  occur.  We  select  0.1  as  our  routine  setting  for  the  Typical  organization 
and  0.2  for  our  Decentralized  and  Holonistic  designs,  reflecting  their  decreasing 
management  potential  for  intervention.  For  stressed  scenarios,  we  use  0.3  for  our 
Typical  organization  and  0.4  for  our  Decentralized  and  Holonistic  designs. 
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